Method and system for automatically detecting morphemes in a task classification system using lattices

ABSTRACT

In an embodiment, a lattice of phone strings in an input communication of a user may be recognized, wherein the lattice may represent a distribution over the phone strings. Morphemes in the input communication of the user may be detected using the recognized lattice. Task-type classification decisions may be made based on the detected morphemes in the input communication of the user.

This application is a continuation of U.S. patent application Ser. No.12/182,618, filed Jul. 30, 2008, which is a continuation of U.S. patentapplication Ser. No. 11/420,738, filed May 27, 2006, which is adivisional of U.S. patent application Ser. No. 10/158,082, filed May 31,2002, which claims the benefit of U.S. Provisional Patent ApplicationNo. 60/322,447, filed Sep. 17, 2001, which is incorporated herein byreference in its entirety. U.S. patent application Ser. No. 10/158,082is also a continuation-in-part of U.S. patent application Ser. Nos.09/690,721 and 09/690,903 (U.S. patent application Ser. No. 09/690,903is now U.S. Pat. No. 6,681,206, issued Jan. 20, 2004) both filed Oct.18, 2000, which claim priority from U.S. Provisional Application No.60/163,838, filed Nov. 5, 1999. U.S. patent application Ser. No.09/690,721, U.S. Pat. No. 6,681,206 and U.S. Provisional Application No.60/163,838 are incorporated herein by reference in their entireties.

TECHNICAL FIELD

The invention relates to automated systems for communication recognitionand understanding.

BACKGROUND OF THE INVENTION

Conventional methods for constructing and training statistical modelsfor recognition and understanding involve collecting and annotatinglarge speech corpora for a task. This speech is manually transcribed andeach utterance is then semantically labeled. The resultant database isexploited to train stochastic language models for recognition andunderstanding. These models are further adapted for different dialogstates. Examples of such methods are shown in U.S. Pat. Nos. 5,675,707,5,860,063, 6,044,337, 6,192,110, and 6,173,261, each of which isincorporated by reference herein in its entirety.

This transcription and labeling process is a major bottleneck in newapplication development and refinement of existing ones. For incrementaltraining of a deployed automated dialog system, current technology wouldpotentially require transcribing millions of transactions. This processis both time-consuming and prohibitively expensive.

SUMMARY OF THE INVENTION

The invention concerns a method and system for detecting morphemes in auser's communication. The method may include recognizing a lattice ofphone strings from an input communication of the user, the latticerepresenting a distribution over the phone strings, detecting morphemesin the input communication of the user using the lattice, and makingtask-type classification decisions based on the detected morphemes inthe input communication of the user.

The morphemes may be acoustic and/or non-acoustic. The morphemes mayrepresent any unit or sub-unit of communication including phones,diphones, phone-phrases, syllables, grammars, words, gestures, tabletstrokes, body movements, mouse clicks, etc. The training speech may beverbal, non-verbal, a combination of verbal and non-verbal, ormultimodal.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is described in detail with reference to the followingdrawings wherein like numerals reference like elements, and wherein:

FIG. 1 is a block diagram of an exemplary task classification system;

FIG. 2 is a detailed block diagram of an exemplary morpheme generator;

FIG. 3 is a flowchart illustrating an exemplary morpheme generationprocess;

FIG. 4 is a flowchart illustrating an exemplary candidate phone-phraseselection process;

FIG. 5 is a flowchart of an exemplary task classification process;

FIG. 6 is a graph illustrating the number of recognized phones perutterance;

FIG. 7 is a graph illustrating the length comparison of recognized vs.transcribed utterances;

FIG. 8 is a graph illustrating the mutual information (MI) ofphone-phrases, showing increased MI as their length increases;

FIG. 9 is a graph illustrating the P_(max) of phone-phrases, showingmore phrases with high P_(max) as their length increases;

FIG. 10 is a graph illustrating the length of salient phone-phrases;

FIG. 11 illustrates examples of salient phone-phrases for “collect”;

FIG. 12 illustrates an example of an acoustic morpheme for “collect”;

FIG. 13 illustrates an example of an acoustic morpheme lattice for“collect”;

FIG. 14 is an exemplary chart illustrating the experimentalnon-detection rates for best path, pruned lattice, and full latticestructures;

FIG. 15 is a graph illustrating the experimental number of detectedacoustic morphemes per sentence;

FIG. 16 is an exemplary chart illustrating the experimental statisticsof a particular acoustic morpheme F_(c);

FIG. 17 is an exemplary chart illustrating the experimental detection ofa particular acoustic morpheme F, given the call-type and its salience;

FIG. 18 is an exemplary chart illustrating the experimental recognitionaccuracy of a particular acoustic morpheme F_(c); and

FIG. 19 is a graph illustrating the experimental call-classificationperformance on speech using acoustic morphemes.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

This invention concerns a dialog system that automatically learns fromspeech without transcriptions. Semantic labels can be extractedautomatically from either experiments or from autonomous dialogs. Inparticular, a task-independent phone-recognizer is taught how to ‘learnto understand’ from a database of untranscribed (or transcribed) speechplus semantic labels.

Baseline approaches to the teaching of a speech recognition systems tounderstand are found in U.S. Pat. Nos. 5,675,707, 5,860,063, 6,044,337,6,192,110 and 6,173,261, which are incorporated herein by reference intheir entireties.

The earliest work demonstrated automatic acquisition of ‘words’ and‘grammar’ from collapsed text. That work did not address, however, theissues arising from non-perfect recognition of speech. The next step wasto show how to acquire lexical units from speech alone withouttranscription and exploit them for spoken language understanding (SLU).That experiment, however, was constrained to speech comprising isolatedword sequences and used matching techniques to decide if an observationwas a new ‘word’ or variation of a known ‘word’.

All of the above efforts involve learning from speech alone. While onecan learn much about a spoken language by merely listening to it, theprocess can progress further and faster by exploiting semantics. Thishas been demonstrated in both the engineering domain and in analyses ofchildren's language acquisition. Thus, this invention goes beyond thepast efforts by exploiting speech plus meaning using morphemes, bothacoustic and non-acoustic, in order to teach a machine to learn tounderstand.

While the morphemes may be non-acoustic (i.e., made up of non-verbalsub-morphemes such as tablet strokes, gestures, body movements, etc.),for ease of discussion, the systems and methods illustrated in thedrawings and discussed in the below concern only acoustic morphemes.Consequently, the invention should not be limited to just acousticmorphemes and should encompass the utilization of any sub-units of anyknown or future method of communication for the purposes of recognitionand understanding.

Furthermore, while the terms “speech”, “phrase” and “utterance”, usedthroughout the description below, may connote only spoken language, itis important to note in the context of this invention, “speech”,“phrase” and “utterance” may include verbal and/or non-verbal sub-units(or sub-morphemes). Therefore, “speech”, “phrase” and “utterance” maycomprise non-verbal sub-units, verbal sub-units or a combination ofverbal and non-verbal sub-units within the spirit and scope of thisinvention.

FIG. 1 is an exemplary block diagram of a possible speech recognitionand understanding system 100 that utilizes acoustic morphemes. Thespeech recognition and understanding system 100 includes two relatedsubsystems, namely a morpheme generation subsystem 110 and input speechclassification subsystem 120.

The morpheme generation subsystem 110 includes a morpheme generator 130and a morpheme database 140. The morpheme generator 130 generatesmorphemes from a corpus of untranscribed training speech (the inventionmay also operate with training speech that is transcribed, of course).The generated morphemes are stored in the morpheme database 140 for useby the morpheme detector 160. The morpheme database 140 contains a largenumber of verbal and non-verbal speech fragments or sub-morphemes(illustrated as phone-phrases for ease of discussion), each of which isrelated to one or more of a predetermined set of task objectives. Eachof the morphemes may be labeled with its associated task objective. Theoperation of the morpheme generator 130 will described in greater detailwith respect to FIGS. 2-4 below.

The input speech classification subsystem 120 includes an input speechrecognizer 150, a morpheme detector 160 and a task classificationprocessor 170. The input speech recognizer 150 receives a user's taskobjective request in the form of verbal and/or non-verbal speech. Theinput speech recognizer 150 may perform the function of recognizing, orspotting the existence of one or more phones, sub-units, acousticmorphemes, etc. in the user's input speech by any algorithm known to oneof ordinary skill in the art. However, the input speech recognizer 150forms a lattice structure to represent a distribution of recognizedphone sequences, such as a probability distribution. The input speechrecognizer 150 may extract the n-best phone strings that may beextracted from the lattice, either by themselves or along with theirconfidence scores. Lattice representations are well known those skilledin the art and are further described in detail below.

While the method of morpheme detection using lattices is shown in thefigures as being associated with a task classification system, this ispurely exemplary. The method of morpheme detection using lattices may beapplied to a wide variety of automated communication systems, includingcustomer care systems, and should not be limited to a taskclassification system.

The morpheme detector 160 then detects the acoustic and/or non-acousticmorphemes present in the lattice that represents the user's inputrequest. The morphemes generated by the morpheme generation subsystem110 are provided as an input to the morpheme detector 160.

The output of morpheme detector 160 includes the detected morphemesappearing in the user's task objective request that is then provided tothe task classification processor 170. The task classification processor170 may apply a confidence function, based on the probabilistic relationbetween the recognized morphemes and selected task objectives, and makesa decision either to implement a particular task objective, or makes adetermination that no decision is likely in which case the user may bedefaulted to a human or automated system for assistance.

An exemplary process of the invention will now be described withreference to FIGS. 2-4. FIG. 2 is a detailed block diagram of anexemplary morpheme generator 130. The morpheme generator 130 includes anASR phone recognizer 210, a salient phone-phrase generator 220, and aclustering device 230.

FIG. 3 illustrates a possible process of generating morphemes using themorpheme generator 130 of FIG. 2. The process begins at step 3000 andproceeds to step 3100 where the ASR phone recognizer 210 receives rawtraining speech from a database, for example. The database may begenerated from recordings of users talking with human agents, respondingto the prompt “AT&T. How may I help you?” (HMIHY). The characteristicsof this data and early experiments are detailed in U.S. Pat. No.5,675,707, for example.

In an embodiment for recognizing non-acoustic morphemes, the ASR phonerecognizer 210 may be replaced in the figure by a sub-morphemerecognizer. The sub-morpheme recognizer would operate similar to the ASRphone recognizer, but it would receive raw non-acoustic or a mixture ofacoustic and non-acoustic training data from a database. However, foreach of discussion, use of only acoustic morphemes will be describedbelow.

In addition, while the drawings illustrate the use of phones, this ispurely exemplary. Any sub-portion of verbal and/or non-verbal speech mayby recognized and detected within the spirit and scope of the invention.

A training set of thousands of spoken utterances with correspondingcall-labels is used, followed by using a separate test set in the rangeof 1000 utterances. They are designated HHS-train and HHS-test,respectively. HHS denotes human/human speech-only.

The ASR phone recognizer 210 that is applied to the training speech istask-independent. In particular, a phonotactic language model wastrained on the switchboard corpus using a Variable-Length N-gramStochastic Automaton. This corpus is unrelated to the HMIHY task, exceptin that they both comprise fluent English speech. Off-the-shelftelephony acoustic models may be used. Applying the ASR phone recognizer210 to the HMIHY test speech data yields a phone accuracy of 43%. Thetraining and test sets so generated are denoted by ASR-phone-train andASR-phone-test respectively.

For a baseline comparison, a ‘noiseless’ phonetic transcription wasgenerated from the orthographic transcriptions, by replacing each wordby its most likely dictionary pronunciation and deletingword-delimiters. E.g. “collect call” is converted to “K ax l eh K T K aol” (see FIGS. 11 and 12, for example). The data sets are denoted astranscr-phone-train and transcr-phone-test.

The number of recognized phones per utterance is distributed as shown inFIG. 6. The mean length is 54 phones per utterance. The shape of thedistribution of time per utterance is similar, with a mean duration of5.9 seconds.

For each utterance, the length of the recognized phone sequence iscompared with the length of the phonetic transcription. These values arescatter-plotted in FIG. 7, with the diagonal shown for reference.Observe that in most cases, the transcribed and recognized utteranceshave approximately the same length. Deviation from the diagonal is intwo directions. The above-diagonal points correspond to speech beingrecognized as background noise or silence. The below-diagonal pointscorrespond to background noise being recognized as speech.

In step 3200, the salient phone-phrase generator 220 selects candidatephone-phrases from the raw training speech corpus. While the system andmethod of the invention is illustrated and described using the termphone-phrases, it is again important to note that phone-phrases areactually sub-morphemes that may be acoustic or non-acoustic (i.e., madeup of non-verbal sub-morphemes such as tablet strokes, gestures, bodymovements, etc.). However, as discussed above, for ease of discussion,the systems and methods illustrated in the drawings and discussed in thebelow concern only phone-phrases. Consequently, the invention should notbe limited to using just phone-phrases and should encompass theutilization of any sub-units of any known or future method ofcommunication for the purposes of recognition and understanding.

FIG. 4 illustrates a more detailed flowchart of the candidatephone-phrase selection process that takes place in step 3200. In step3210, the raw training speech corpus is filtered using grammaticalinference algorithms, such as those defined in U.S. Pat. No. 5,675,707.As a result of the filtering process, in step 3220, all observed phonesequences of the predetermined length are selected. In step 3230, hesalient phone-phrase generator 220 determines whether the desiredmaximum phrase length has been met. If the desired maximum phrase lengthhas been met, in step 3240, the salient phone-phrase generator 220selects the phone sequence as a candidate phone-phrase. Conversely, ifthe desired maximum phrase length has not been met, the process returnsto step 3210 to resume filtering the training speech corpus.

Once the candidate phone-phrases have been selected, in step 3300, thesalient phone-phrase generator 220 selects a subset of the candidatephone-phrases. Thus, new units are acquired by the above process ofsearching the space of observed phone-sequences and selecting a subsetaccording to their utility for recognition and understanding. Theresultant subset selected is denoted as salient phone-phrases. Examplesof salient phone-phrases for the word “collect” are shown in FIG. 11.

The salient phone-phrase generator 220 may perform the selection ofsalient phone-phrases by first using a simplified measure of thecandidate phone-phrase's salience for the task as the maximum of the aposteriori distribution,

${{P_{m\;{ax}}(f)} = {\max\limits_{C}{\Pr\left( C \middle| f \right)}}},$

where C varies over the 15 call-types in the HMIHY task. The salientphone-phrases are then selected by applying a threshold on P_(max) andby using a multinomial statistical significance test. This significancetest excludes low-frequency phrases for which a fortunate conjunction ofevents can give a high appearance salience purely by chance. It teststhe hypothesis that the observed call-type count distribution is asample from the prior distribution.

In step 3400, the salient phone-phrases are clustered into acousticmorphemes by the clustering device 230. FIG. 12 shows an example ofacoustic morpheme containing the word “collect”. The clustering isachieved using a combination of string and semantic distortion measuresusing methods, for example, such as those in U.S. Pat. No. 6,173,261.Each cluster is then compactly represented as a finite state machine andstored in the acoustic morpheme database 140. The acoustic morphemesstored in the acoustic morpheme database 140 may then be used in thetask classification process shown in FIG. 5.

The example below illustrates this acoustic morpheme generation process.Consider a candidate phone-phrase,f=p ₁ p ₂ . . . p _(n),

where p_(i) are phones. Denote its frequency by F(f). A measure of itsutility for recognition is the mutual information of its components,denoted MI(f), which may be approximated viaMI(f)=MI(p ₁ p ₂ . . . p _(n-1) ;p _(n)).

As discussed above, a simplified measure of its salience for the task isthe maximum of the a posteriori distribution,

${{P_{{ma}\; x}(f)} = {\max\limits_{C}{\Pr\left( C \middle| f \right)}}},$

where C varies over the 15 call-types in the HMIHY task.

These features for phone-phrases observed in the noise-free case arecharacterized transcr-phone-train. In FIG. 8, the MI distributions ofthese phone-phrases are shown for lengths 2-4. It can be observed thatthe MI distributions shift positively as length increases, corroboratingthe increased predictive power of longer units. It can also be observed,however, that while many of these phrases have positive predictive power(MI>0), many do not. Thus, for constructing larger units from smallerones, attention is restricted to the positive tail of these MIdistributions.

For each of these phone-phrases, p_(max)(f) is computed, which is ameasure of the salience of a phrase for the task. FIG. 9 shows thedistribution of P_(max) for varying length phrases. It can be observedthat for single phones, P_(max) is near-random, corroborating theintuition that semantics is carried by longer phone-phrases. It can alsobe observed that the positive shift in the distributions as lengthincreases. In particular, focus on the region P_(max)>0.9, whichcorresponds to highly salient phone-phrases. As length increases, moreof these highly salient phrases are discovered.

The goal of this process is to grow the phone-phrases until they havethe salience of words and word-phrases. Thus, the search criteria forselecting longer units is a combination of their utility forwithin-language prediction, as measured by MI, and their utility for thetask, as measured by P_(max). For phrases with large P_(max)., the MI ofthe phrase tends to be larger than average. This correlation wasexploited successfully for frequency-compensated salience in earlierexperiments discussed above, but is not exploited here. In the earlierexperiments, a set of salient phone-phrases of length≦16 was generatedvia a two-pass process as follows:

Select phone-phrases with F(f)≧5 and length≦4;

Filter the training corpus ASR-phone-train with those phrases, using aleft-right top-down filter with the phrases prioritized by length.

Select subsequences from the filtered corpus of fragment-length≦4, (i.e.with #phones≦16) and with MI≧1 and P_(max)≧0.5.

This particular iterative selection process was selected based on easeof implementation should not be considered optimal. The resultant set ofsalient phone-phrases have lengths≦16, distributed as shown in FIG. 10.

FIG. 5 is a flowchart of a possible task classification process usingacoustic morphemes. The process begins at step 5000 and proceeds to step5100 where input speech recognizer 150 receives an input communication,such as speech, from a user, customer, etc. The input speech may, ofcourse, be expressed in verbal speech, non-verbal speech, multimodalforms, or using a mix of verbal and non-verbal speech.

Non-verbal speech may include but are not limited to gestures, bodymovements, head movements, non-responses, text, keyboard entries, keypadentries, mouse clicks, DTMF codes, pointers, stylus, cable set-top boxentries, graphical user interface entries and touchscreen entries, or acombination thereof. Multimodal information is received using multiplechannels (i.e., aural, visual, etc.). The user's input communication mayalso be derived from the verbal and non-verbal speech and the user's orthe machine's environment. Basically, any manner of communication fallswithin the intended scope of the invention. However, for ease ofdiscussion, the invention will be described with respect to verbalspeech in the examples and embodiments set forth below.

Working with the output of the non-perfect input speech recognizer 150introduces the problem of the coverage of the user's input with theacoustic morphemes. To counter this problem, in step 5200, a latticestructure is formed to improve the coverage of the recognizer 150.Lattices are efficient representations of a distribution of alternativehypothesis. A simple example of a lattice network, resulting from theutterance “collect call”, is shown in FIG. 13, where a bold circlerepresents an initial state and a double circle a final state. The mostlikely phone sequence of the transcribed utterance is “K ax l eh K T Kao I”, and the best path of the input speech phone recognizer 150 is “KI ah K ao l”. Whereas the salient phone-phrase “K ax I eh K T K ao I” isnot present in the best path, it does appear in the lattice network.Exploiting lattices results in additional matches of the salient phrasesin the utterances, as compared to searching only in the best paths.

In speech recognition, the weights (likelihoods) of the paths of thelattices are interpreted as negative logarithms of the probabilities.For practical purposes, the pruned network will be considered. In thiscase, the beam search is restricted in the lattice output, byconsidering only the paths with probabilities above a certain thresholdrelative to the best path. The threshold r is defined as: r_(i)≦r, withr_(i)=p_(i)/p₁, where p_(i) is the probability of the i^(th) path and p₁is the probability of the best path.

In order to quantify the coverage, the number of test sentences arefirst measured with no detected occurrences of acoustic morphemes, forthe experiments using best paths, pruned lattices and full lattices.These numbers are illustrated in FIG. 14. Observe that 42% of the bestpath sentences have no detected acoustic morphemes. When the search ofthe acoustic morphemes is expanded to the pruned lattices, the number ofsentences with no detections decreases to 12%. This number drops down to6% when full lattices are used for the search.

The relative frequency distributions of the number of detected acousticmorphemes in the best paths, in the pruned lattices, and in the fullnetwork experiments, are shown in FIG. 15. As expected, the number ofdetections increases in the experiments using lattices.

As shown above, expanding the search of the acoustic morphemes in thelattice network results in improved coverage of the test sentences. Itis of course accompanied by an increased number of false detections ofthe acoustic morphemes. In order to study in more detail the falsedetection issue, it is beneficial to focus on one particular morphemeF_(c), strongly associated to the call-type c=collect. Its occurrencesin the best paths will be compared with its occurrences in the latticenetwork. The salient phone-phrases clustered in this morpheme representvariations of the phrase “collect call”. A subgraph of this morpheme wasshown in FIG. 12. Its salience on the training set is P(c/F_(c))=0.89.The following notation: W={manually chosen word sequences correspondingto F_(c)}. is introduced.

FIG. 16 shows the comparison of the coverage of the test utterances withthe acoustic morpheme F_(c), on the best paths and on the latticenetwork. F_(c) is detected in 3% of the best paths. Searching in thefull lattice network, increases this coverage to 8%. However, intranscr-word-test, the word sequences corresponding to the AcousticMorpheme F_(c), are present in only 7% of the transcribed sentences.

FIG. 17 illustrates the relationship between the detections of F_(c)given the call-type, P(F_(c)|c) and its salience, P(c|Fc), measured onthe test set. As shown in the table, the number of detections of thismorpheme given the call-type increases from 15% in the case of the bestpaths, to 31% in the case of the full lattice search. In parallel, adecrease in the salience from 93% to 71% is observed. This is anindication that the high salience of this morpheme on the best paths isconserved in the pruned lattices, but not in the full lattices.

FIG. 18 illustrates the recognition accuracy of the Acoustic MorphemeF_(c), as compared to the transcribed text. The probabilities P(F_(c)|W)and P(F_(c)| W) indicate how often the acoustic morpheme is found in theASR phone output, given that it is known that the corresponding wordsequences are present (or not) in the transcribed sentences. Searchingin the full lattice increases the number of detected occurrences ofF_(c) from 38% up to 66%, albeit with a parallel increase of the falselydetected morphemes.

In step 5250, the acoustic morpheme detector 160 detects acousticmorphemes that have been recognized and formed into a lattice structureby the input speech recognizer 150 using the acoustic morphemes storedin the acoustic morpheme database 140. In step 5300, the taskclassification processor 170 performs task classifications based on thedetected acoustic morphemes. The task classification processor 170 mayapply a confidence function based on the probabilistic relation betweenthe recognized acoustic morphemes and selected task objectives, forexample. In step 5400, the task classification processor 170 determineswhether a task can be classified based on the detected acousticmorpheme. If the task can be classified, in step 5700, the taskclassification processor 170 routes the user/customer according to theclassified task objective. The process then goes to step 5900 and ends.

If the task cannot be classified in step 5400 (i.e. a low confidencelevel has been generated), in step 5500, a dialog module (locatedinternally or externally) the task classification processor 170 conductsdialog with the user/customer to obtain clarification of the taskobjective. After dialog has been conducted with the user/customer, instep 5600, the task classification processor 170 determines whether thetask can now be classified based on the additional dialog. If the taskcan be classified, the process proceeds to step 5700 and theuser/customer is routed in accordance with the classified task objectiveand the process ends at step 5900. However, if task can still not beclassified, in step 5800, the user/customer is routed to a human forassistance and then the process goes to step 5900 and ends

An experiment evaluating the utility of these methods in the HMIHY taskwas conducted. A classifier was trained from the output of a phonerecognizer on 7462 utterances, which was denoted ASR-phone-train.Salient phone-phrases of length≦16 were selected, as described above.The salient phone-phrases were then clustered into salient grammarfragments. A single-layer neural net was trained with these fragments asinput features. The resultant classifier was applied to the 1000utterance test-set, ASR-phone-test.

The call-classification results are scored following the methodology ofU.S. Pat. No. 5,675,707. In this method, an utterance is classified bythe system as one of 14 call-types or rejected as ‘other’. Rejection isbased on a salience-threshold for the resulting classification. Onedimension of performance is the False Rejection Rate (FRR), which is theprobability that an utterance is rejected in the case that the userwanted one of the call-types. The cost of such an error is a lostopportunity for automation. The second dimension of performance is theProbability of Correct Classification (P_(c)) when the machine attemptsa decision. The cost of such an error is that of recovery via dialog.Varying the rejection threshold traces a performance curve with axesP_(c) and FRR.

Searching in the lattice network will introduce the additional problemof multiple detections of the acoustic morphemes on different levels ofthe lattice network, and the issue of combining them optimally. For anevaluation of the usefulness of the phone lattices, the problem oftreating multiple detections will be deferred from different levels ofthe network, and stop our search in the lattice network as soon as asentence with one or more detections is found. The existing call-typeclassifier is modified in the following way: for the test sentenceswithout detected occurrences of the acoustic morphemes in the bestpaths, the search is expanded in the lattice network and stops as soonas a sentence with one or more detections is found.

FIG. 19 shows the results of call-classification experiments trained onword transcriptions, and exploiting spoken language understanding withutterance verification compared with the utility of the detectedoccurrences of the acoustic morphemes in the pruned lattice testsentences. Using this new lattice-based detection method, an operatingpoint with 81% correct classification rate is achieved at rank 2, with15% false rejection rate. This is a reduction of 59% from the previousfalse rejection rate using best paths, albeit with a 5% reduction in thecorrect classification performance from that baseline.

As shown in FIGS. 1 and 2, the method of this invention may beimplemented using a programmed processor. However, method can also beimplemented on a general-purpose or a special purpose computer, aprogrammed microprocessor or microcontroller, peripheral integratedcircuit elements, an application-specific integrated circuit (ASIC) orother integrated circuits, hardware/electronic logic circuits, such as adiscrete element circuit, a programmable logic device, such as a PLD,PLA, FPGA, or PAL, or the like. In general, any device on which thefinite state machine capable of implementing the flowcharts shown inFIGS. 3-5 can be used to implement the speech recognition andunderstanding system functions of this invention.

While the invention has been described with reference to the aboveembodiments, it is to be understood that these embodiments are purelyexemplary in nature. Thus, the invention is not restricted to theparticular forms shown in the foregoing embodiments. Variousmodifications and alterations can be made thereto without departing fromthe spirit and scope of the invention.

1. A method comprising: receiving verbal speech and non-verbal speechfrom a user; selecting verbal candidate morphemes from the verbal speechand non-verbal candidate morphemes from the non-verbal speech; forming,via a processor, a first lattice, based, at least in part, on the verbalcandidate morphemes and stored morphemes, and forming a second latticebased on the non-verbal candidate morphemes; determining a taskobjective of the speech, based at least in part on the first lattice andthe second lattice; and routing the user according to the taskobjective.
 2. The method of claim 1, further comprising: determining asecond task objective of the non-verbal speech, based at least in parton the non-verbal lattice; and determining if one of the task objectiveand the second task objective has priority.
 3. The method of claim 1,wherein the non-verbal speech comprises at least one of gestures, bodymovements, head movements, non-responses, text, keyboard entries, keypadentries, mouse clicks, DTMF codes, pointers, stylus, cable set-top boxentries, graphical user interface entries, and touchscreen entries. 4.The method of claim 1, wherein the verbal candidate morphemes areexpressed in multimodal form.
 5. The method of claim 1, furthercomprising prompting the user to provide a feedback response.
 6. Themethod of claim 5, wherein the feedback response comprises at least oneof a confirmation with respect to the task objective and additionalinformation with respect to the speech.
 7. The method of claim 1,wherein the first lattice determines a salience measure associated withthe task objective and the verbal candidate morphemes.
 8. A systemcomprising: a processor; and a non-transitory computer-readable storagemedium having stored therein instructions which, when executed by theprocessor, cause the processor to perform a method comprising: receivingverbal speech and non-verbal speech from a user; selecting verbalcandidate morphemes from the verbal speech and non-verbal candidatemorphemes from the non-verbal speech; forming a first lattice based, atleast in part, on the verbal candidate morphemes and stored morphemes,and forming a second lattice based on the non-verbal candidatemorphemes; determining a task objective of the speech, based at least inpart on the first lattice and the second lattice; and routing the useraccording to the task objective.
 9. The system of claim 8, theinstructions, when executed by the processor, cause the processor toperform a method further comprising: determining a second task objectiveof the non-verbal speech, based at least in part on the non-verballattice; and determining if one of the task objective and the secondtask objective has priority.
 10. The system of claim 8, wherein thenon-verbal speech comprises at least one of gestures, body movements,head movements, non-responses, text, keyboard entries, keypad entries,mouse clicks, DTMF codes, pointers, stylus, cable set-top box entries,graphical user interface entries, and touchscreen entries.
 11. Thesystem of claim 8, wherein the verbal candidate morphemes are expressedin multimodal form.
 12. The system of claim 8, the instructions, whenexecuted by the processor, cause the processor to perform a methodfurther comprising prompting the user to provide a feedback response.13. The system of claim 12, wherein the feedback response comprises atleast one of a confirmation with respect to the task objective andadditional information with respect to the speech.
 14. The system ofclaim 8, wherein the first lattice determines a salience measureassociated with the task objective and the verbal candidate morphemes.15. A non-transitory computer-readable storage medium storinginstructions which, when executed by a computing device, cause thecomputing device to perform a method comprising: receiving verbal speechand non-verbal speech from a user; selecting verbal candidate morphemesfrom the verbal speech and non-verbal candidate morphemes from thenon-verbal speech; forming a first lattice based, at least in part, onthe verbal candidate morphemes and stored morphemes, and forming asecond lattice based on the non-verbal candidate morphemes; determininga task objective of the speech, based at least in part on the firstlattice and the second lattice; and routing the user according to thetask objective.
 16. The non-transitory computer-readable storage mediumof claim 15, the instructions, when executed by the computing device,cause the computing device to perform a method further comprising:determining a second task objective of the non-verbal speech, based atleast in part on the non-verbal lattice; and determining if one of thetask objective and the second task objective has priority.
 17. Thenon-transitory computer-readable storage medium of claim 15, wherein thenon-verbal speech comprises at least one of gestures, body movements,head movements, non-responses, text, keyboard entries, keypad entries,mouse clicks, DTMF codes, pointers, stylus, cable set-top box entries,graphical user interface entries, and touchscreen entries.
 18. Thenon-transitory computer-readable storage medium of claim 15, wherein theverbal candidate morphemes are expressed in multimodal form.
 19. Thenon-transitory computer-readable storage medium of claim 15, theinstructions, when executed by the computing device, cause the computingdevice to perform a method further comprising prompting the user toprovide a feedback response.
 20. The non-transitory computer-readablestorage medium of claim 19, wherein the feedback response comprises atleast one of a confirmation with respect to the task objective andadditional information with respect to the speech.